Papers by Jonathan K. Kummerfeld
Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review (2025.acl-long)
Copied to clipboard
| Challenge: | Clinical coding is labour-intensive and error-prone, which has motivated research towards full automation of the process. |
| Approach: | They propose to use AI to improve evaluation methods and propose new methods to assist clinical coders in their workflows. |
| Outcome: | The proposed methods can be improved and improved on existing methods and the existing ones to assist coders in their workflows. |
A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. |
| Approach: | They propose a multi-stage crowd workflow that reduces expert involvement without sacrificing accuracy. |
| Outcome: | The proposed method reduces expert effort by 4x, from 56% to 14% of cases. |
Compositional Demographic Word Embeddings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are usually derived from corpora containing text from many individuals . however, they cannot account for user-specific word preferences, such as using the same word in different ways across contexts. |
| Approach: | They propose a new form of personalized word embeddings that use demographic-specific word representations derived compositionally from full or partial demographic information for a user. |
| Outcome: | The proposed representations outperform generic representations on two English language tasks. |
Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing (2021.acl-short)
Copied to clipboard
| Challenge: | Existing research suggests that crowd workers need to complete a substantial amount of poorly paid work to earn a fair wage. |
| Approach: | They propose to use a qualification that requires workers to have completed a certain number of tasks to earn a fair wage. |
| Outcome: | The proposed qualification reduces the burden on workers while still collecting high quality data. |
Improving Text-to-SQL Evaluation Methodology (P18-1)
Copied to clipboard
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev
| Challenge: | Current evaluations of text-to-SQL systems are limited by the way they divide data into training and test sets. |
| Approach: | They propose to standardize and improve existing and new text-to-SQL datasets . they propose a template-based slot-filling baseline that cannot generalize to new queries . |
| Outcome: | The proposed system is competitive with prior work on multiple datasets and can be used on training and test sets. |
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (D19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars
| Challenge: | Task-oriented dialog systems need to know when a query falls outside their range of supported intents. |
| Approach: | They propose a dataset that includes queries that are out-of-scope and 150 intent classes over 10 domains. |
| Outcome: | The proposed dataset includes queries that are out-of-scope, i.e., queries that do not fall into any of the system’s supported intents. |
Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods (2020.coling-main)
Copied to clipboard
| Challenge: | Standard slot-filling models train or finetune on large datasets of carefully-annotated data that is domain specific. |
| Approach: | They propose automatic methods to identify inconsistencies in crowd-annotated data . a slot-filling model can extract the tokens "New York" as a TO LOCATION slot in a query . |
| Outcome: | The proposed methods reveal inconsistencies in data, though there is scope for improvement. |
Exploring the Value of Personalized Word Embeddings (2020.coling-main)
Copied to clipboard
| Challenge: | a subset of words belonging to specific psycholinguistic categories vary more in their representations across users . combining generic and personalized word embeddings yields the best performance . |
| Approach: | They propose personalized word embeddings and compare their performance to generic ones . they show that personalized word representations can be leveraged for improved performance . |
| Outcome: | The proposed model can be used for authorship attribution. |
Factors Influencing the Surprising Instability of Word Embeddings (N18-1)
Copied to clipboard
| Challenge: | Word embeddings are low-dimensional, dense vector representations that capture semantic properties of words. |
| Approach: | They examine the stability of word embeddings by examining their properties and analyzing their effects on downstream tasks. |
| Outcome: | The results show that even high frequency words exhibit substantial instability, which can have implications for downstream tasks. |
Exploring Self-Identified Counseling Expertise in Online Support Forums (2021.findings-acl)
Copied to clipboard
Allison Lahnala, Yuntian Zhao, Charles Welch, Jonathan K. Kummerfeld, Lawrence C An, Kenneth Resnicow, Rada Mihalcea, Verónica Pérez-Rosas
| Challenge: | Increasing number of people engage in online health forums, making it important to understand the quality of the advice they receive. |
| Approach: | They examine the role of expertise in responses to help-seeking posts . they find that a classifier can distinguish between peer and self-identified mental health professionals' interactions . |
| Outcome: | The findings show that experts' language use differs between groups, and that their comments engage the support-seeker further. |
Improving Low Compute Language Modeling with In-Domain Embedding Initialisation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train language models on in-domain data are limited. |
| Approach: | They propose to initialise and freeze in-domain embeddings to provide a useful representation of rare words in English . they find that the standard configuration is not optimal when rare words are present . |
| Outcome: | The proposed approach improves language modeling by providing a useful representation of rare words in English. |
Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Health (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing statistical models are not explainable, struggle in low-resource scenarios and cannot be reused for multiple tasks. |
| Approach: | They propose a micromodel architecture that embeds domain knowledge and provides explanations throughout the model’s decision process. |
| Outcome: | The proposed model is validated on depression classification, PTSD classification, and suicidal risk assessment tasks. |
SLATE: A Super-Lightweight Annotation Tool for Experts (P19-3)
Copied to clipboard
| Challenge: | a new annotation tool is designed to fill the niche of a lightweight interface for terminal users . current tools are built with direct manipulation via a Graphical User Interface (GUI) this approach is time-consuming and difficult to modify . |
| Approach: | They propose a terminal-based annotation tool that supports multiple annotations . they use a text-based interface that uses almost the entire screen to display documents . |
| Outcome: | The proposed tool is designed to fill the niche of a lightweight interface for users with a terminal-based workflow. |
Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness (2020.emnlp-main)
Copied to clipboard
Stefan Larson, Anthony Zheng, Anish Mahendran, Rishi Tekriwal, Adrian Cheung, Eric Guldan, Kevin Leach, Jonathan K. Kummerfeld
| Challenge: | Recent work on dialog has found that crowdsourced data can have limited diversity as workers tend to write simple variations from prompts. |
| Approach: | They propose a general approach for guiding workers to write more diverse text by iteratively constraining their writing. |
| Outcome: | The proposed approach improves performance on dialog tasks and improves on existing datasets. |
Outlier Detection for Improved Data Quality and Diversity in Dialog Systems (N19-1)
Copied to clipboard
Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, Jason Mars
| Challenge: | Existing methods to detect outliers in text have been neglected in NLP . outlier detection is a problem in dialog systems where text is often no more than a few sentences in length. |
| Approach: | They propose a technique that uses sentence embeddings to detect outliers in short texts using neural sentence embeds and distance-based outlier detection. |
| Outcome: | The proposed technique detects outliers in a corpus of short texts while generating highly diverse corpora that produce more robust intent classification and slot-filling models. |
World Knowledge for Abstract Meaning Representation Parsing (L18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsers are based on annotated graphs, but there is still room for improvement . |
| Approach: | They examine the role played by world knowledge in parsing errors in a state-of-the-art parser . they examine the effects of different types of world knowledge on parsers . |
| Outcome: | The proposed model improves on multiple fine-grained metrics, including a 6% increase in named entity F-score, and provides insight into the potential of world knowledge for future work in Abstract Meaning Representation parsing. |
Should I Trust You? Detecting Deception in Negotiations using Counterfactual RL (2025.findings-acl)
Copied to clipboard
Wichayaporn Wongkamjan, Yanze Wang, Feng Gu, Denis Peskoff, Jonathan K. Kummerfeld, Jonathan May, Jordan Lee Boyd-Graber
| Challenge: | Future human-AI interaction tools can build on our methods for deception detection by triggering friction to give users a chance to interrogate suspicious proposals. |
| Approach: | They propose to use CTRL-D to detect deception in a board game called Diplomacy . CTRL is a counterfactual RL that has a good recall and almost perfect precision . future tools could build on this to reevaluate trust in suspicious negotiations . |
| Outcome: | The proposed method detects human deception with a high precision when compared to a Large Language Model approach that flags many true messages as deceptive. |
Your Students Don’t Use LLMs Like You Wish They Did (2026.acl-long)
Copied to clipboard
| Challenge: | Educational NLP systems are evaluated using engagement metrics and satisfaction surveys . authors identify a fundamental misalignment between pedagogical design and student usage patterns . |
| Approach: | They propose a computational framework for measuring behaviour in student-AI dialogue . they validate their framework by analysing 12,650 messages from four courses . |
| Outcome: | The proposed metrics outperform surveys and satisfaction surveys on student-AI dialogues. |
Analyzing the Surprising Variability in Word Embedding Stability Across Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Word embeddings are powerful representations that form the foundation of many natural language processing architectures. |
| Approach: | They explore word embedding stability in a wide range of languages to gain insight into their stability. |
| Outcome: | The proposed results provide insights into word embedding stability in English and other languages. |
A Large-Scale Corpus for Conversation Disentanglement (P19-1)
Copied to clipboard
Jonathan K. Kummerfeld, Sai R. Gouravajhala, Joseph J. Peper, Vignesh Athreya, Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros C Polymenakos, Walter Lasecki
| Challenge: | a dataset of 77,563 messages manually annotated with reply-structure graphs disentangles conversations and defines internal conversation structure. |
| Approach: | They use a dataset of 77,563 messages manually annotated with reply-structure graphs to disentangle conversations and define internal conversation structure. |
| Outcome: | The new dataset is 16 times larger than all previous datasets combined and includes adjudication of annotation disagreements and context. |
Personalized Help for Optimizing Low-Skilled Users’ Strategy (2025.naacl-short)
Copied to clipboard
Feng Gu, Wichayaporn Wongkamjan, Jordan Lee Boyd-Graber, Jonathan K. Kummerfeld, Denis Peskoff, Jonathan May
| Challenge: | a natural language agent generates moves and messages based on player intentions . a dozen games with novice and experienced players generate useful advice . |
| Approach: | a team of researchers augment a natural language agent to generate move and message advice . they use a game to simulate the intentions of novice and experienced players . |
| Outcome: | The enhanced agent generates move and message advice based on player intentions . the agent helps novices compete with experienced players and even surpass them . |
Effective Crowdsourcing for a New Type of Summarization Task (N18-2)
Copied to clipboard
| Challenge: | Currently, summarization research focuses on summarizing the entire text, but in practice, readers are often interested in only one aspect of the document or conversation. |
| Approach: | They propose a new task where the goal is to summarize a particular aspect of a document. |
| Outcome: | The proposed task is based on a crowdsourced data collection workflow that allows users to collect high-quality summaries. |
Using Paraphrases to Study Properties of Contextual Embeddings (2022.naacl-main)
Copied to clipboard
| Challenge: | Previously, paraphrases have been used to probe whether compositionality is accurately captured by BERT, but we believe they can be used to explore many other questions. |
| Approach: | They propose to use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT. |
| Outcome: | The proposed analysis of paraphrases and paraphrase representations using the Paraphrase Database shows that BERT handles polysemous words, but different representations in many cases. |
Less is More: Explainable and Efficient ICD Code Prediction with Clinical Entities (2025.acl-long)
Copied to clipboard
| Challenge: | Clinical coding is labor-intensive and prone to delays, leading to global backlogs. |
| Approach: | They propose an approach that combines Named Entity Recognition (NER) and Assertion Classification (AC) to filter for clinically important content before supervised code prediction. |
| Outcome: | The proposed approach reduces training time by over half on a standard evaluation dataset compared to current methods . it uses Named Entity Recognition (NER) and Assertion Classification (AC) to filter for clinically important content before supervised code prediction. |
Leveraging Similar Users for Personalized Language Modeling with Limited Data (2022.acl-long)
Copied to clipboard
| Challenge: | Recent work suggests that personalized models are more accurate for individual users than one-size-fits-all solutions. |
| Approach: | They propose a model trained on users that are similar to a new user to find similarity between new and existing users. |
| Outcome: | The proposed model can predict what a user will write when they join a platform and not enough text is available. |
Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate SQL from natural language are still making many mistakes . a new interaction mechanism allows users to edit a step-by-step explanation of a query to fix errors. |
| Approach: | They propose a mechanism that allows users to edit a step-by-step explanation of a query to fix errors. |
| Outcome: | The proposed approach can achieve better performance than multiple SOTA approaches on multiple datasets and 24 participants. |
Data Collection for Dialogue System: A Startup Perspective (N18-3)
Copied to clipboard
| Challenge: | Developing dialogue systems such as Apple Siri and Google Now requires high quality training data but data collection with crowdsourcing is largely an open question. |
| Approach: | They propose to use crowdsourcing to collect data for a user intent classification task in a dialogue system. |
| Outcome: | The proposed method improves the quality of the collected data and the model performance on real user queries. |